The new features of AbiPy v0.9.1¶

M. Giantomassi and the AbiPy group¶

10th international ABINIT developer workshop
May 31 - June 4, 2021 - Smart Working, Lockdown

  • These slides have been generated using jupyter, nbconvert and revealjs
  • The notebook can be downloaded from this github repo
  • To install and configure the software, follow these installation instructions

Use the Space key to navigate through all slides.

What is AbiPy?¶

Python package for:¶

  • Generating ABINIT input files automatically
  • Post-processing output results (netcdf and text files)
  • Interfacing ABINIT with external tools
  • Generating and executing ABINIT-specific workflows

Dependencies:¶

  • Hard deps: numpy, scipy, pandas, matplotlib, netcdf4, plotly, sympy, pymatgen
  • Soft deps: ipython, jupyter, phonopy, ASE.

NB: AbiPy requires pymatgen but it can be interfaced with other packages (e.g ASE, phonopy, etc) via converters

What's new at the level of the installation procedure?¶

  • AbiPy now requires python >= 3.6
  • conda packages for AbiPy and ABINIT are now provided by conda-forge
  • AbiPy v1.0 will drop support for Abinit8
  • We plan to provide recipes for EasyBuild and Spack
  • We seek for volunteers to support other package managers (homebrew, apt-get, etc)

How to install AbiPy¶

Using conda and the conda forge channel (recommended):

    conda install abipy --channel conda-forge

Recipes for AbiPy and ABINIT are now supported by the conda-forge community thus:

  • conda packages are automatically generated when new ABINIT/AbiPy releases are pushed to github
  • the abiconda channel is deprecated. Please use the conda-forge versions

Since conda is not limited to py packages, one can install ABINIT in the same env with:

    conda install abinit -c conda-forge

Using pip and python wheels:

    pip install abipy --user

From the github repository:

    git clone https://github.com/abinit/abipy.git
    cd abipy 
    python setup.py install # or `develop` for development

For further info see http://abinit.github.io/abipy/installation.html

What's new at the level of the documentation?¶

  • New website based on Read-the-Docs theme
  • Gallery of plots with matplotlib and plotly examples (XXX scripts in total)
  • New examples for Flows (XXX scripts in total)
  • Each example can be executed inside a Docker repo on mybinder.org
  • New examples in the ABINIT EPH tutorials.

New AbiPy website based on Read-the-Docs theme¶

In [3]:
%embed https://abinit.github.io/abipy/index.html
Out[3]:

Sphinx-gallery with binder button to run the example in a Docker container¶

In [4]:
%embed https://abinit.github.io/abipy/gallery/plot_phbands_nkpt_tsmear.html#sphx-glr-gallery-plot-phbands-nkpt-tsmear-py
Out[4]:
In [5]:
#%embed https://abinit.github.io/abipy/gallery/index.html
In [6]:
#%embed https://abinit.github.io/abipy/flow_gallery/index.html

What's new at the level of workflow machinery?¶

  • Python converters DDB $\;\rightleftarrows \;$ phonopy / tdep

  • New Flows and post-processing tools for:

    • elastic and piezoelectric tensors with DFPT (clamped/relaxed atoms)
    • Gruneisen parameters with DFPT + finite differences
    • quasi-harmonic approximation with DFPT
    • effective masses with DFPT or finite differences
    • isotropic supercoducting $T_c$ with Mc Millan equation
    • e-ph self-energy, e-ph matrix elements and scattering potentials
    • phonon-lmited transport properties

in addition to the standard workflows for GS, DFPT, $GW$, IP optics, Bethe-Salpeter, etc.

Two different workflows infrastructures:¶

Internal implementation (abipy.flowtk Flows):¶

  • ✅ Lightweigth, no database is required
  • ✅ Designed for rapid prototyping and for testing advanced ABINIT capabilities
  • ❌ No explicit support for high-throughput (HT) applications.

AbiFlows package (requires Fireworks and MongoDB database)¶

  • ✅ HT-oriented: use database to store workflow status for further analysis
  • ✅ High-level no-brainer API designed for HT applications
  • ❌ New ABINIT features are first implemented/tested with flowtk Flows and then ported to AbiFlows.

NB: There's an ongoing effort to reimplement AbiFlows in terms of the atomate framework. In this talk will mainly discuss the new features implemented in abipy.flowtk.

DDB converters¶

(contributed by G. Petretto)

  • Based on py script by A. Romero. ...

  • Applications:

    • Interface AbiPy with phonopy
    • Connect ABINT DPT with other requiring phonony (e.g. anharmonic calculations hipive)

Example: obtain irreps from DDB using phonopy

ElasticWork¶

This flow computes:¶

  • the rigid-atom elastic tensor
  • the rigid-atom piezoelectric tensor (insulators only)
  • the internal strain tensor
  • the atomic relaxation corrections to the elastic and piezoelectric tensor

For the formalism, see:

Python API:¶

scf_input = make_scf_input() # Build input for GS calculation

elast_work = flowtk.ElasticWork.from_scf_input(scf_input, 
                                               with_relaxed_ion=True, 
                                               with_piezo=True)

To generate a GS input, one can use factory functions:¶

In [8]:
from abipy.abio.factories import gs_input
gs_inp = gs_input(structure="si.cif", 
             pseudos="14si.pspnc",
             kppa=None, ecut=None, pawecutdg=None, scf_nband=None, accuracy="normal", spin_mode="polarized",
             smearing="fermi_dirac:0.1 eV", charge=0.0, scf_algorithm=None)
gs_inp
Out[8]:
##############################################
#### SECTION: basic
##############################################
ecut 0.0
ngkpt 8 8 8
shiftk 0.5 0.5 0.5
nshiftk 1
kptopt 1
nsppol 2
nband 16
occopt 3
tolvrs 1e-08
##############################################
#### SECTION: gstate
##############################################
spinat
0.0 0.0 0.6
0.0 0.0 0.6
chksymbreak 0
nspinor 1
nspden 2
charge 0.0
tsmear 0.0036749322175655 Ha
##############################################
#### STRUCTURE
##############################################
natom 2
ntypat 1
typat 1 1
znucl 14
xred
0.0000000000 0.0000000000 0.0000000000
0.2500000000 0.2500000000 0.2500000000
acell 1.0 1.0 1.0
rprim
6.3285005244 0.0000000000 3.6537614813
2.1095001748 5.9665675141 3.6537614813
0.0000000000 0.0000000000 7.3075229627

or build the GS input explictly if fine tuning is needed:¶

In [9]:
multi = abilab.ebands_input(structure="si.cif", 
                            pseudos="14si.pspnc",
                            ecut=8, 
                            spin_mode="unpolarized", 
                            smearing=None, 
                            dos_kppa=5000)

multi.get_vars_dataframe("kptopt", "iscf", "ngkpt")
Out[9]:
kptopt iscf ngkpt
dataset 0 1 None [8, 8, 8]
dataset 1 -11 -2 None
dataset 2 1 -2 [14, 14, 14]

Dependency Graph:¶

To compute elastic tensors from the DDB file, use:¶

In [10]:
elastic_ddb = abilab.abiopen("elastic_DDB")
edata = elastic_ddb.anaget_elastic()

edata.elastic_relaxed
Out[10]:
xx yy zz yz xz xy
Voigt index
xx 135.262182 54.450376 38.052927 0.00000 0.000000 0.000000
yy 54.450376 135.262181 38.052927 0.00000 0.000000 0.000000
zz 38.052927 38.052926 148.211029 0.00000 0.000000 0.000000
yz 0.000000 0.000000 0.000000 30.55071 0.000000 0.000000
xz 0.000000 0.000000 0.000000 0.00000 30.550709 0.000000
xy 0.000000 0.000000 0.000000 0.00000 0.000000 40.405903

To compute elastic properties (rigid/relaxed atoms)¶

In [31]:
edata.get_elastic_properties_dataframe(properties_as_index=True)
Out[31]:
property 0 1
0 trans_v 3.194459e+03 3.838052e+03
1 long_v 5.796035e+03 6.295200e+03
2 snyder_ac 5.767044e+01 8.693459e+01
3 snyder_opt 3.158141e-01 3.621134e-01
4 snyder_total 5.798626e+01 8.729670e+01
5 clarke_thermalcond 7.734051e-01 9.006348e-01
6 cahill_thermalcond 8.539415e-01 9.791319e-01
7 debye_temperature 3.760623e+02 4.477874e+02
8 k_voigt 7.553865e+01 7.553930e+01
9 k_reuss 7.553074e+01 7.553579e+01
10 k_vrh 7.553469e+01 7.553754e+01
11 g_voigt 3.951341e+01 5.767416e+01
12 g_reuss 3.761300e+01 5.366047e+01
13 g_vrh 3.856321e+01 5.566731e+01
14 universal_anisotropy 2.527308e-01 3.740360e-01
15 homogeneous_poisson 2.818554e-01 2.041909e-01
16 y_mod 9.886491e+10 1.340681e+11
For additional info, please consult this notebook tutorial¶

Gruneisen parameters¶

This flow computes:¶

  • $\gamma_{\bf{q}\nu}$ using finite differences and DFPT phonons

Python API:¶

scf_input = make_scf_input() # Build input for GS calculation

# NB: k-mesh in gs_inp and ngqpt q-mesh must be commensurate.
from abipy.flowtk.gruneisen import GruneisenWork
voldelta = gs_inp.structure.volume * 0.02
work = GruneisenWork.from_gs_input(gs_inp, voldelta, ngqpt=[2, 2, 2], with_becs=False)
  • Three relaxations at fixed volume (violet task)
  • Each relaxation task starts a DFPT computation of the dynamical matrix.

To analyze the results with python, use:¶

In [13]:
gruns = abiopen("out_GRUNS.nc")
gruns.plot_gruns_scatter();

abiopen.py out_GRUNS.nc --expose-web #

Non-linear optical properties¶

(contributed by F. Naccarato)

This flow computes:¶

  • SCF + NSCF along k-path to find band edges automatically
  • Perform DFPT computation of $m^*$ for these $k$-points
  • See J. Abreu's talk for applications

For the formalism see:

Python API:¶

scf_input = make_scf_input()   # Get the SCF input (without SOC)

scf_input = make_scf_input()
return flowtk.NonLinearCoeffFlow.from_scf_input("nlo_flow", scf_input)

Effective masses with DFPT¶

This flow computes:¶

  • SCF + NSCF along k-path to find band edges automatically
  • Perform DFPT computation of $m^*$ for these $k$-points
  • See J. Abreu's talk for applications

For the formalism see:

Python API:¶

scf_input = make_scf_input()   # Get the SCF input (without SOC)

from abipy.flowtk.effmass_works import EffMassAutoDFPTWork
flow = flowtk.Flow("effmass_flow")

work = EffMassAutoDFPTWork.from_scf_input(scf_input)
flow.register_work(work)
  • SCF + NSCF along k-path to find band edges automatically
  • Perform DFPT computation of $m^*$ for these $k$-points
  • See Joao's talk

e-ph matrix elements along an arbitrary q-path¶

This flow computes:¶

This example shows how to compute the e-ph matrix elements in AlAs along a q-path with AbiPy flows. The final results are stored in the GKQ.nc file (one file for q-point) in the outdata of each task.

Python API:¶

# Build input for GS calculation on a 2x2x2 k-mesh
scf_input = make_scf_input(ngkpt=(2, 2, 2))

# q-mesh for phonons
ngqpt = (2, 2, 2)

# Create flow to compute all the independent atomic perturbations
# Use ndivsm = 0 to pass an explicit list of q-points.
# If ndivsm > 0, qpath_list is interpreted as a list of boundaries for the q-path
qpath_list = [[0.0, 0.0, 0.0], [0.01, 0, 0], [0.1, 0, 0],
              [0.24, 0, 0], [0.3, 0, 0], [0.45, 0, 0], [0.5, 0.0, 0.0]]

from abipy.flowtk.eph_flows import GkqPathFlow
flow = GkqPathFlow.from_scf_input("flow_dir", scf_input,
                                  ngqpt, qpath_list, ndivsm=0, with_becs=True,
                                  ddk_tolerance={"tolwfr": 1e-8})

Formalism:

https://abinit.github.io/abipy/flow_gallery/run_gkq.html#sphx-glr-flow-gallery-run-gkq-py

  • SCF + NSCF along k-path to find band edges automatically
  • Perform DFPT computation of $m^*$ for these $k$-points
  • See Joao's talk

Other implementations/applications of AbiPy flows are discussed in the following talks:¶

  • G.Brunin
  • Gabriel's PhD student
  • Joao
  • Julien

What's new at the level of the post-processing tools?¶

  • New plotting tools based on plotly
  • GUIs and dashboards based on panel and bokeh

Why plotly?¶


  • ✅ Publication quality figures.
  • ✅ Flexible python API able to produce rather advanced plots
  • ❌ Plots are difficult to customize without changing the python code
  • ❌ Plots lacks interactivity and integration with HTML/JS



  • ✅ Interactive plots + chart editor GUI to customize the figure
  • ✅ Play well with HTML (plotly us written in js with python bindings)
  • ❌ Open source project but not all the features are available in the free plan
  • ❌ Requires browser (this may represent an issue on some HPC centers)

Important note¶

  • AbiPy will continue to support and develop matplotlib-based tools
  • We plan to implement plotly-based tools for the most important physical properties but we cannot support all the possible plots already implemented with matplotlib
  • Fortunately, there are packages such as holoviz or plotly-express that provided the data is stored in a pandas dataframe.

Several other plotting libraries are available in the pydata ecosystem

AbiPy plots with matplotlib¶

In [15]:
gsr = abiopen("si_nscf_GSR.nc")
gsr.ebands.plot(with_gaps=True);

AbiPy plots with plotly¶

(contributed by Y. He and MG)

In [16]:
gsr.ebands.plotly(with_gaps=True);  # object.plot becomes object.plotly

To update the plotly figure to the chart studio server, use:¶

gsr.ebands.plotly(with_gaps=True, chart_studio=True);

Users can finally customize the AbiPy plot without changing the py code.¶

Interactive 3d plots with plotly:¶

In [18]:
gsr.ebands.kpoints.plotly();

The matplotlib 3d plot embedded in HTML :¶

In [19]:
gsr.ebands.kpoints.plot();

Integrating Abipy with web-based technologies¶

  • Integration between panel and AbiPy
  • AbiPy GUIs inside jupyter notebooks
  • Dashboards and web apps
  • Integration with the AbiPy command line interface

What is Panel?¶

  • Panel provides tools for composing widgets, plots, tables into apps and dashboards
  • It relies on the client-server model where:
    • the client is the web browser running HTML/CSS/JS code.
    • the server communicates with the client, executes py code and sends the results back to the client
  • In a nutshell, panel keeps the browser and python in synch e.g. the user clicks a button in the GUI and the signal to python.

Why panel?¶

  • Panel works with visualizations from Bokeh, Plotly, Matplotlib, HoloViews, and many other Python plotting libraries,
  • Panel works equally well in Jupyter Notebooks

Thanks to this infrastructure, developers can¶

  • use python to solve the scientific problem using the pydata ecosystem and other ab-inition py packages
  • use modern web-technologies to present the results to the client without having to deal with HTML/CSS/JS

Pros and cons of the client-server model¶

Advantages:¶

  • Client does not need to install the scientific software stack (when running on different machines)
  • Can take advantage of modern HTML technologies to present the results in the browser
  • Can implement web apps that allows the user to upload the data and analyze the results withouth having to install python software.

Disavantages¶

  • Round trip delay if client != host and slow connection
  • Upoloadng a 1Gb file to the remote server just because you don't want to install software on the localhost is a very bad idea.
  • OK for relatively small files (< 1Gb) but this approach is not designed to handle big data.
  • Not all the HPC centers provide specialized nodes to post-process the results inside a web browser/notebook.

How to use AbiPy panels inside jupyter notebooks¶

To build a panel GUI inside the notebook, invoke the get_panel method:

In [20]:
#gsr = abiopen("si_nscf_GSR.nc")
#abilab.abipanel();
#gsr.get_panel()
In [21]:
ddb = abilab.abiopen("ZnSe_hex_qpt_DDB")

abilab.abipanel(); # Important
ddb.get_panel()
Out[21]:
Don't be surprised if you start to click buttons and **nothing happens**! One needs a **running python backend** to execute the callbacks triggerered by the GUI widgets.

Other AbiPy files/objects are supported as well:¶

In [22]:
ddb.structure.get_panel()
Out[22]:

Creating dashboards from the command line¶

  • Using get_panel inside a notebook is great if you need both the flexibility of the python language and the easiness of the GUIs to improve your productivity.
  • In some cases, however, we just want to create a dahhboard with widgets to interact with the data.

To produce a predefined set of matplotlib figures, use:¶

abiopen.py mgb2_kpath_FATBANDS.nc --expose --seaborn

abiopen.py mgb2_kpath_FATBANDS.nc --expose-web # -ewb

abiopen.py mgb2_kpath_FATBANDS.nc --plotly # -ply

abiopen_expose

To create a panel dashboard associated to the DDB file, use:

abiopen.py out_DDB --panel  # or -pn if you prefer the short version.

Creating dashboards without widgets with abiview¶

to get a quick look at the results

abiview.py ddb ZnSe_hex_qpt_DDB --panel  # or -pn if you prefer the short version.

Documentation for these new features available at:¶

In [23]:
%embed https://abinit.github.io/abipy/graphical_interface.html
Out[23]:
In [25]:
#plotter = abilab.ElectronBandsPlotter()
#plotter.add_ebands(label="BZ sampling", bands="si_scf_GSR.nc")
#plotter.add_ebands(label="k-path", bands="si_nscf_GSR.nc")
#plotter.gridplot(with_gaps=True);

Future developments¶

Post-processing tools¶

  • Implement panel GUIs for all the netcdf files already supported by AbiPy
  • Develop web applications so that users can upload e.g. DDB files and analyze the results

Continuous Integration¶

  • Use AbiPy programmatic interface to implement:

    • Validation of parallel algorithms for np in range(1, N)
    • Stress testing
    • Benchmarks